Skip to content

tests+docs(design): the period-honorific rows declare tolerated; W2 keeps its normative half - #505

Merged
derek73 merged 5 commits into
masterfrom
claude/period-class-demotion
Sep 5, 2026
Merged

tests+docs(design): the period-honorific rows declare tolerated; W2 keeps its normative half#505
derek73 merged 5 commits into
masterfrom
claude/period-class-demotion

Conversation

@derek73

@derek73 derek73 commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Three CJK case rows whose text ends in an ASCII period after an honorific — 田中さん 様., 김민준 씨., 김민준 양. — declare tolerated and leave the contract tier, closing a gap in the 2026-09-01 demotion's sweep criterion. Refs #469 (the demotion this completes) and #501 (one of its four contests changes severity).

What was wrong

The 2026-09-01 sweep demoted every CJK-bearing row carrying a comma or an ASCII letter. A trailing ASCII period is neither, so three fix(#320) probe rows — the spaced twins of comma rows that were demoted — stayed contract, and one of them was then chosen as W2's replacement example because it passed the same test. A period after a CJK honorific is a listing artifact no writing system produces: exactly the class the doctrine demotes. And W2's clause "the split-off ignores surrounding punctuation" had no pure witness, because native writing has no punctuation around a name to ignore — which makes it descriptive, not a promise.

Measured 2026-09-05 with the wider criterion (any non-space ASCII in a CJK-bearing untolerated row): eight texts. The three period rows are this PR. 山田 太郎 (マイケル・ジャクソン) is kept: the parser reads ASCII, fullwidth and corner-bracket delimiters identically, and what the row pins is the nakaguro inside a delimited nickname. The four mixed-script rows (John 王, 王·Smith, Andersonさん, Anderson선생님) were reviewed and kept by d562164, for reasons that still hold.

What changes

  • Rows and validator. The three rows carry tolerated=True with the reason in their notes; they still pin the A trailing period stops a CJK honorific being recognized ("김민준, 씨." → given "씨.") #320 mechanism at HEAD. Shape 6/7 purity now refuses any ASCII character other than a space (letters and commas already were; punctuation and digits join them); untagged rows are untouched, so the mixed-script rows stay where d562164 put them. corpus_cjk.jsonl 73 → 70, corpus_cjk_tolerated.jsonl 26 → 29, the contract floor lowered deliberately.
  • W2 → W3, half of it. W2's sentence had two halves. The punctuation half ("ignores surrounding punctuation") had no pure witness and moves to W3 with its 田中さん 様. example. The other half — a part that is not name text is never the name's end, so the split-off steps past a post-nominal word and never dissects it — is normative and stays in W2, now witnessed by pure contract-tier names (김민준 박사님 → suffix 박사님; 선생님 as the boundary): forcing that step off in memory dissects 박사님 into 박사, 님 and 선생님 into three, the Glued CJK honorifics are absorbed into the name (山田太郎様 → family 山田太郎様) #308/Unspaced Chinese/Korean names: surnames constants + longest-match segmentation #271 regression the case rows pin. The branch's first draft took both halves; the design-docs review caught it, and decisions.md records the correction. corpus_rules.jsonl 248 → 247 → 249, every arrival already in corpus_cjk.jsonl. The example line moved in the first commit, not the second: test_the_tolerated_corpus_is_disjoint_from_the_contract_ones refuses a text that is demoted and still enforced, so no half-demoted state exists on the branch.
  • The period is not ignored; it is decisive. W3's descriptive text says what is measured rather than what the old clause claimed: the step past a post-nominal word is W2's and needs no punctuation (田中さん 様. divides where 田中さん 様 does), while a period glued to the honorific's own word is read and decides against the split-off — 田中さん. and 김민준씨. each parse as a title — and 田中さん 太郎. versus 田中さん 太郎 flips the order. Five parses in the decisions entry.
  • Records. decisions.md#cjk-comma-demotion gains a dated supersession note (the criterion was narrower than the doctrine it implemented; the widened rule; the nickname keep with its delimiter measurement; the clause move). The _WATCHED_DIFFS Six contested diffs in the 2.x ledgers have no pin, and the empty rosters read as if none existed (MD, PHD goes to whichever rule comes first) #501 block's "three contract-tier" becomes two, and the watched-shapes arc's measurement paragraph is corrected to match.

Verification

Intentional counts 352 / 247 / 155 / 14 unmoved at every commit (a demotion moves nothing, and the two W2 witnesses were already pooled), 0 unexplained, 0 radar-unclassified, no finding block. Mutation control: 田中さん 様.'s watched row corrupted now prints MOVED SHAPE (radar) and exits 0 where master exited 1, and 선생님 corrupted identically still exits 1 — the severity followed the tier, and the check is live. Full suite green; ruff and mypy clean. Re-running the sweep after: 8 → 5 texts, the period class gone, the letter class untouched.

🤖 Generated with Claude Code

derek73 and others added 5 commits September 5, 2026 13:10
…6/7 refuse ASCII punctuation

The 2026-09-01 CJK demotion swept the case table for a comma or an
ASCII letter. A trailing ASCII period is neither, so three fix(#320)
probe rows -- '田中さん 様.', '김민준 씨.' and '김민준 양.', each the
spaced twin of a comma row that WAS demoted -- stayed on the contract
tier. A period standing after a CJK honorific is a listing artifact no
writing system produces, which is exactly the class that demotion
names; the rows carry tolerated=True now.

The criterion follows the doctrine rather than sampling it: a shape 6/7
tag admits no ASCII character but the space between two name words, so
a period, a digit and a bracket are refused beside the comma and the
Latin letter already refused. That gate is a property of the shape TAG,
not of CJK text -- an untagged row may carry ASCII, and the four
mixed-script rows d562164 reviewed by name keep the contract tier
untouched. Measured 2026-09-05 by a wider sweep (CJK-bearing untolerated
rows carrying any non-space ASCII, grouped by kind): eight texts before
this commit, five after -- the period class gone, the letter class
whole, and '山田 太郎 (マイケル・ジャクソン)' deliberately kept, its
ASCII parentheses being one spelling of a delimiter the writing system
does produce.

corpus_cjk.jsonl 73 -> 70 and corpus_cjk_tolerated.jsonl 26 -> 29, the
identity holding at 99 either side. _CORPUS_FLOORS lowered 70 -> 67
deliberately; the tolerated floor of 22 still stands under 29.

W2's example line for '田中さん 様.' goes with the flag rather than with
the clause it witnessed, which lands in the next commit: the plan held
it back, but test_the_tolerated_corpus_is_disjoint_from_the_contract_ones
refuses a text that is tolerated in the case table while a contract
corpus still holds it -- the "demoted and still enforced" state nobody
reads for. So corpus_rules.jsonl 248 -> 247 here, and the demotion is
complete at this commit rather than half-made. W2 keeps the clause and
loses its witness for one commit.

Gate at four baselines: 352 / 247 / 155 / 14 intentional, unexplained 0,
radar unclassified 0, exit 0, no MOVED SHAPE and no NOT CHECKED.
compare.py's #501 block comment still calls '田中さん 様.' contract-tier
and is stale from here; it is re-measured two commits on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lerated zone

W2 read "The split-off ignores surrounding punctuation, but never
treats a part that is not name text as the name's end". Native CJK
writing puts no punctuation around a name for a split-off to ignore, so
every input the clause describes is a listing form -- W3's ground, and a
description rather than a promise. The clause had no pure witness and
could not be given one: its example, '田中さん 様.', left in the
previous commit with the flag, and the two boundary lines that might
have looked like witnesses for the second half ('김지양', '王君') are
about the vocabulary's own limits, not about punctuation. The second
half was already stated at more length in W3 ("A part that is nothing
but suffix words is not the name's end: it is declined as the site"), so
both halves went.

W3 gains the example line, one descriptive sentence, and a rationale
clause naming the punctuation a listing carries in beside the comma
itself; its `tolerated:` marker now counts five illustrated-not-promised
lines where it counted four. W2 keeps its glued-honorific statement and
five examples, none carrying ASCII -- measured 2026-09-05, W2's line was
the only CJK example outside W3 that carried an ASCII character at all.
Both `interacts:` lines still hold: W2 points at W3, where the crossing
and now the punctuation reading live.

_script_segment.py's W2 citation is trimmed to the surviving statement
in the same commit (test_citations_are_verbatim_excerpts catches it) and
says where the punctuation half went.

corpus_rules.jsonl is unchanged at 247: build_rules_corpus.py skips a
tolerated rule's examples whole, so the line arriving in W3 adds nothing
back. decisions.md#cjk-comma-demotion takes the record -- the criterion
the 2026-09-01 sweep used, the wider one that replaces it, the
parenthesized-nickname row kept and the delimiter measurement behind
that keep, and the recompute for the sweep itself.

Gate at four baselines: summary lines byte-identical to the previous
commit's -- 352 / 247 / 155 / 14 intentional, unexplained 0, radar
unclassified 0, exit 0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The four #501 contest rows closing the 2.0.0 watched section carried
"Three are contract tier (corpus_cjk.jsonl; the first also
corpus_rules.jsonl), so a move on them fails the run". Two are, now:
'김민준 박사님' and '선생님'. '田中さん 様.' left both contract files in
this bundle's first two commits and reads radar, so a shape move on it
prints and no longer fails.

That is the thing worth stating and not the digit: a row's fatality
follows the tier its NAME reads at, which no roster in this file
records, so a demotion elsewhere changes what a row here costs without
touching the row -- the comment is the only thing that goes stale, and
only a re-measurement finds it.

Mutation controls, in memory around main() with the worktree untouched,
2026-09-05, and run as a PAIR because either alone is consistent with a
dead check:

  '田中さん 様.' corrupted at 2.0.0 --
  MOVED SHAPE (radar) expected_since_2.0.0.toml: 1 watched diff
  shape(s) disagree with this run. [...] This does not fail the run
  because the name is radar tier (_CORPUS_TIERS), which watches
  without promising.
  main() returns 0. It returned 1 before this bundle.

  '선생님' corrupted the same way --
  MOVED SHAPE expected_since_2.0.0.toml: [...] This fails the run
  because the name is contract tier (_CORPUS_TIERS).
  main() returns 1, so the exit 0 above is the tier and not a check
  that stopped running.

_CORPUS_TIERS's "no text in the tolerated file is held by a contract
corpus" sentence is re-verified and gains what the demotion taught it:
'田中さん 様.' was one of the two PURE texts the 2026-09-01 W2 swap
brought in, so marking its case row was not enough on its own -- the
example line had to move into the tolerated W3 in the same breath, which
test_the_tolerated_corpus_is_disjoint_from_the_contract_ones enforces on
every run. decisions.md's watched-shapes measurement, taken hours before
the demotion, is corrected in the same commit for the same claim.
tools/differential/README.md needed no edit: its "three contract-tier
names" sentence is about the #498 rule-pair swap, a different three.

Full suite 6507 passed, ruff clean, mypy clean over 110 files. Gate at
four baselines: 352 / 247 / 155 / 14 intentional, unexplained 0, radar
unclassified 0, exit 0, no MOVED SHAPE and no NOT CHECKED.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nored, it is decisive

W2's clause read "The split-off ignores surrounding punctuation, but
never treats a part that is not name text as the name's end". The
previous commit moved BOTH halves to W3 on a rationale that reaches
only the first, and the second half is normative: it is what the
parser does with no comma and no period anywhere in the input, and
contract-tier pure names depend on it.

1. W2's SECOND HALF RESTORED, with pure witnesses. Measured 2026-09-05
by forcing _is_post_nominal to False in memory -- the predicate the
site scan _peel_site reads -- and re-parsing (family, given, suffix,
title): '田中さん 様' ships ('田中', '', 'さん, 様', '') and gives
('田中さん', '', '様', '') without the step; '김민준 박사님' ships
('김', '민준', '박사님', '') and gives ('김', '민준', '박사, 님', '');
'선생님' ships ('선생님', '', '', '') and gives ('선', '생', '님', '').
The post-nominal word is dissected in every case. '김민준 박사님' and
'선생님' are corpus_cjk.jsonl names on the CONTRACT tier, and
ko_honorific_glued_doctor_spaced is the #308/#271 regression row for
the first -- under W2's remaining statement alone, 박사님 ends in the
listed tail 님 and reads as peelable. The sentence goes back in W2's
register and takes two witnesses whose texts corpus_cjk.jsonl already
held: '김민준 박사님' -> suffix="박사님" and the boundary line '선생님'
-> family="선생님". '田中さん 様' would have been the third and is NOT
added -- it is not a case row, and a new contract name needs a 1.4.0
classification this bundle does not have.

2. W3's NEW DESCRIPTIVE SENTENCE REWRITTEN to what is measured. It
attributed the crossing to the punctuation; the punctuation does not
license it and is not ignored either. W3 now says the step past a
post-nominal word is W2's and fires with no punctuation present, that
a period on a SEPARATE post-nominal word rides into the suffix and
moves no division, and that a period glued to the honorific's own word
decides against the split-off. The '田中さん 様.' example line stays;
'田中さん.' is not a case-row text, so it is described in prose rather
than given an example line.

3. decisions.md#cjk-comma-demotion. The moved-clause note is rewritten
as a within-branch correction: the FIRST half had no pure witness and
moved to W3 as description, the SECOND is normative, pure-witnessed
and was cut in error. "Each the spaced twin of a comma row that WAS
demoted" is corrected -- '田中さん, 様.' and '김민준, 씨.' are rows,
'김민준, 양.' is not; enumerating the CASES texts containing 양 gives
김지양, 양 미선, 김민준 양, 김민준 양. and 양 지훈, and '김민준 양.'
is demoted on the class argument alone. A FOUND bullet records the
period measurement: '田中さん 様' -> family 田中, suffix 'さん, 様';
'田中さん 様.' -> family 田中, suffix 'さん, 様.'; '田中さん 太郎.' ->
family '太郎.', given '田中さん' (family '田中さん', given '太郎'
without the period -- the order flips); '田中さん.' -> title
'田中さん.'; '김민준씨.' -> title '김민준씨.'. The one input where a
period changes nothing is the one where it sits on a word the
split-off steps past regardless.

4. The watched-shapes quiet-tree figure is PINNED to the commit it was
measured at. This branch's 214a1ed touched _script_segment.py, so
a965272..HEAD is 6 / 41 / 16 where the paragraph claims 5 / 38 / 15 --
which is a965272..4dd776a, still true there. Recorded as a point
measurement with the next touching commit named.

5. Two stale enumerations gain the third demoted class, matching
cases.py's wording: compare.py's _CORPUS_TIERS comment and
build_cjk_corpus.py's module docstring both listed only the comma
listing and the Latin title/credential.

6. The three period rows' notes drop the repeated "the 2026-09-01
sweep's 'comma or ASCII letter' criterion could not see" -- provenance
about a past oversight, stated three times where the decisions note
states it once. The constraint half and the #320-at-HEAD half stay.

_script_segment.py carries the restored sentence as a second W2
citation, on _peel_site where the scan-back lives, and the
_peel_honorific_tail block is corrected to what W3 now says.

corpus_rules.jsonl 247 -> 249, both arrivals already in
corpus_cjk.jsonl, so the compared pool is unchanged at 1113 / 1120.
Full suite 6509 passed, 0 xpass; ruff clean; mypy clean over 110
files. Gate at four baselines: 352 / 247 / 155 / 14 intentional,
unexplained 0, radar unclassified 0, exit 0, no MOVED SHAPE and no NOT
CHECKED. All four generated corpora regenerate byte-identical.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, and W3 says what pins it

Five prose corrections to what this bundle's own review round wrote, each
re-measured on this tree at d0faed4.

The recorded-shapes snapshot contrasted its pinned quiet-tree figure with a
live one "at HEAD", and that live figure was already stale at the commit
that wrote it -- the exact defect the pin exists to avoid. The contrast is
removed rather than refreshed: the pinned 5 / 38 / 15 over a965272..4dd776a
is now the only number, and the paragraph says instead that 214a1ed and
d0faed4 touched _script_segment.py's comments and W2/W3 citations, so a HEAD
recompute is larger and moves with every further comment-only commit. The
RECOMPUTE line now insists on the pinned endpoint.

The W2 bullet's summary of its three forced parses said "in every case the
post-nominal word is dissected". Re-run with _is_post_nominal rebound to
False, only two are: 박사님 -> 박사, 님 and 선생님 -> 선, 생, 님. The first,
田中さん 様, yields ('田中さん', '', '様', '') -- 様 whole, but taken as the
split site, so it breaks the other clause, never the name's end. The bullet
now names which clause each case witnesses.

The same bullet had the polarity of its own argument backwards, saying a W2
reduced to its first sentence "would have stopped contradicting" the
regression its example pins. parse('김민준 박사님').suffix is '박사님'; under
the first sentence alone 박사님 ends in the listed tail 님 and reads as
peelable, so the reduced W2 would have BEGUN contradicting
ko_honorific_glued_doctor_spaced -- which is the argument for restoring the
half, not against it.

The FOUND bullet said "four parses" over a list its own later sentences call
"the five inputs": four counts the sentences, one of which carries two
strings. It is five now, by input, with the convention stated and the sixth
string (the no-period twin 田中さん 太郎) accounted for.

W3's new period sentence asserted that 田中さん. and 김민준씨. each read as a
title, and neither string is a case row or in any corpus, so nothing pins
those readings. W3 is tolerated and illustrates rather than promises, and a
new name would need a baseline classification this bundle does not carry, so
no rows are added; the sentence says instead that it reports parses measured
2026-09-05 that no row pins and that can move unannounced, pointing at
decisions.md#cjk-comma-demotion. _script_segment.py's comment restating the
same reading is aligned to say the same.

Verified: 698 passed on the doc/rules/decisions/mechanisms/guard/citation
subset, ruff clean, and the differential at --baseline 2.2.0 exits 0 with 14
intentional diffs, 0 unexplained, 0 radar unclassified.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.66%. Comparing base (4dd776a) to head (88b025b).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #505   +/-   ##
=======================================
  Coverage   98.66%   98.66%           
=======================================
  Files          45       45           
  Lines        3219     3219           
=======================================
  Hits         3176     3176           
  Misses         43       43           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@derek73
derek73 merged commit e004ccf into master Sep 5, 2026
11 checks passed
@derek73
derek73 deleted the claude/period-class-demotion branch September 5, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation fixes and updates tests tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant